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ABSTRACT 



This study addresses the validation of a holistic English 
assessment process used at Cabrillo College (California) . The process of 
prompt development, scoring rubrics, reader training, scoring, and 
administration procedures have remained largely unchanged over the years. 

This report provides new evidence on content validity, consequential and cut 
score validity, reliability, test bias review, and disproportionate impact. 

To provide data for the validation study, 18,398 English assessments between 
fall 1994 and fall 2000 were used. Information was derived from the CAPP 
system and Cabrillo' s MIS database. Procedures and standards of the 
California Community College Chancellor's Office for assessment instrument 
validation provided the framework for the analyses in this report. A 
representative committee of three English faculty reviewed the assessment 
test for content validity. For each course content item, reviewers checked 
off the essay grading criteria that would test for that course content item. 
All 3,932 essays from fall 1999 to fall 2000 were reviewed for interrater 
reliability. In general, Cabrillo' s assessment appears to place students with 
reasonable accuracy. Proportions of placements were significantly different 
between ethnicities and between genders. About three-quarters of Cabrillo 
assessed students felt they were appropriately qualified for the English 
class in which they were enrolled. Logistic regressions were run to determine 
the interaction of educational background and demography. (JA) 
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Introduction 

Cabrillo College has maintained a locally managed holistic English assessment process. This 
process previously received full approval from the Chancellor’s office and is now due for validation 
renewal. The process of prompt development, scoring rubrics, reader training, scoring, and 
administration procedure have remained largely unchanged. Current prompts are presented in Table 
1 and current scoring rubrics are attached. This report provides new evidence on content validity, 
consequential and cut score validity, reliability, test bias review, and disproportionate impact. 

Table 1. English Assessment Prompts 

1. We constantly get advice from others about the way we should live. Describe a piece of 

advice you have received that you agree or disagree with and use an example or experience 
to show why you agree or disagree with it. 

2. Write about a turning point in your life. 

3. Our first impression of a person often turns out to be inaccurate. Describe how your 

impression of a person improved after you got to know him or her, 

4. Descnbe a past experience (mental or physical), which has taught you something about 

your values, expectations, or beliefs. 



Methods 

18,398 English assessments between Fall 1994 and Fall 2000 provided data for the validation 
study. Information was derived from the CAPP system and Cabrillo’s MIS database. Procedures 
and standards of the California Community College Chancellor’s Office for assessment instrument 
validation provided the framework for the analyses in this report. Data were analyzed with SPSS 
10.0.7. 

Content Validity 

A representative committee of three English faculty reviewed the assessment test for content 
validity. The committee was provided with the prompts, anchor papers, reader norming documents, 
and target course outlines. They were given a matrix (see attachment) with essay grading criteria on 
one axis and course content for target courses on the other axis. For each course content item, 
reviewers checked off the essay grading criteria that would test for that course content item. English 
has three target courses for the assessment process: English 290, a basic skills course; English 100, a 
pre-transfer level course; and English 1A, a transfer level course. Each member of the committee 
found that every course content item was covered by at least one essay grading criterion. The high 
degree of agreement between the test and course content is expected as the faculty who teach the 
target. courses designed our assessment. 
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Table 2. English Assessment Content Validity 



Eligibility 

for: 


Measured Unmeasured 
Prerequisites Prerequisites 


Unrelated 

Criteria 


% Faculty 
Agreement 


English 100 


6 


0 


0 


100 


English 1A 


7 


0 


0 


100 



Reliability 

All 3,932 essays from Fall 1999 to Fall 2000 were reviewed for interrater reliability. All essays 
are initially reviewed by two reviewers and in the case of a tie, a third reader makes the 
determination. Readers generally consist of English faculty and instructional assistants. Readers 
receive training on the scoring rubric including anchor papers that are examples of writing at a given 
level. Periodic norming sessions are conducted to promote interreader consistency especially when a 
new reader has been added to the pool. Readers are typically unaware of other readers 5 scores but 
additional measures could be taken to ensure this is the case. The initial two readings place students 
into the same class 90.9% of the time and agree exacdy 81.7% of the time and showed a 0.83 
correlation indicating an adequate reliability. 

Consequential and Cut Score Validity 

To assess the accuracy and validity of placement, randomly selected English sections were given 
surveys midway through the Fall 2000 semester. Students rated on a 5 point scale how qualified 
they felt for the class they were in. Likewise, instructors in these sections rated on a 5-point scale 
how likely they felt each student was to pass their class. Only data from those students who enrolled 
through Cabrillo's assessment process are presented here. 392 Cabrillo assessed students responded 
to the survey and instructors rated 337 Cabrillo assessed students. In general, Cabrillo’s assessment 
appears to place students with reasonable accuracy. About three-quarters of Cabrillo assessed 
students felt they were appropriately qualified for the English class in which they were enrolled 
(Table 3). Almost 90% of Cabrillo assessed students were rated as able to pass their class by their 
instructors (Table 4). The assessment process appeared to work best for placement into the highest 
level of English. The lowest level has a number of students who do not feel prepared. This concern 
has been echoed by some of the English faculty who feel that a portion of students in basic skills 
English would benefit from taking an ESL or adult literacy class before attempting English 290. 
However, enrollment in basic skills classes cannot be restricted and assessment processes by law 
cannot direct these students to alternative courses. 
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Table 3. English Assessees* Rating of their Preparedness for their English class. 







Student Rating 
Not Qualified Qualified 


Overqualified 


Total 


ENGL-290 


n 


14 


55 


7 


76 




% 


18.4 


72.4 


9.2 


100 


ENGL- 100 


n 


33 


180 


37 


250 




% 


13.2 


72.0 


14.8 


100 


ENGL-1A 


n 


4 


57 


5 


66 




% 


6.1 


86.4 


7.6 


100 


Total 


n 


51 


292 


49 


392 




% 


13.0 


74.5 


12.5 


100.0 



Table 4. Instructor Rating of Possibility of Success of English Assessees 






Instructor Rating 
Will Fail Will Pass 


Total 


ENGL-290 


n 


8 


65 


73 




% 


11.0 


89.0 


100 


ENGL-100 


n 


24 


183 


207 




% 


11.6 


88.4 


100 


ENGL-1A 


n 


2 


55 


57 




% 


3.5 


96.5 


100 


Total 


n 


34 


303 


337 




% 


10.1 


89.9 


100 



Disproportionate Impact 

Placement rates of students were compared between ethnic groups, genders, learning disability 
groups, and ages. Proportions of placements were significantly different between ethnicities and did 
not meet the Chancellor’s Office 80% standard, which states that the relative frequency of one 
ethnicity should be at least 80% of all other ethnicities’ relative frequencies (Table 5). The difference 
are most striking between the two predominant groups, Whites and Hispanics, with Whites being 
3.4 times more likely to be placed in transfer level English than Hispanics. Proportions of 
placements were significantly different between genders and did not meet the Chancellor’s Office 
80% standard (Table 6). Females were 1.6 times more likely than males to place into transfer level 
English. Reasons for this difference are currently unknown. Proportions of placements were 
significantly different between those with learning disabilities and those without and did not meet 
the Chancellor’s Office 80% standard (Table 7). Those without known learning disabilities were 2.9 
times more likely to place into transfer level English than those who identified themselves as 
learning disabled. An explanation for this finding is currently not known. There was a statistically 
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significant but practically small difference in the mean ages of placement groups with a slight trend 
of older students placing in more advanced classes (Table 8). 

Causative factors for these impacts are unknown but may be related to differences in educational 
backgrounds or socioeconomic status that vary with demographics. One possible causal factor 
explored was high school origin as high schools in the area differ in demographics and may differ in 
instructional quality so that what appears to be differences between ethnicities may actually reflect 
differences in the quality of high school instruction. To test this possibility, placement rates between 
ethnicities within groups who attended the same high school were compared. The overall placement 
patterns were replicated within each high school so that it appears that differences in placement rates 
between ethnicities cannot be adequately explained by high school origin. 

At Cabrillo, those who place into lower classes are significantly less likely to be high school 
degree holders, have fewer years of English course work, have lower self-reported English grades, 
and lower overall high school GPA’s. Compared to Whites, Hispanics are significandy less likely to 
report having a high school diploma, to have taken fewer years of English, lower grades in their last 
English class, and lower high school GPA’s. Learning disabled students are significandy less likely 
to have high school diplomas, report lower grades in their last English class, and report lower high 
school GPA’s. Females report significandy higher grades in their last English class and overall highs 
school GPA’s. The oldest 25% of students are significandy less likely to report having high school 
diplomas and have had fewer years of English. Further, while socioeconomic data is not routinely 
collected but we see that those placing in the lower levels were significandy more likely to apply for 
financial aid, which may be taken as an indicator of socioeconomic status that may be related to 
access to pre-collegiate educational opportunities (Table 9). 

Multinomial logistic regression shows that a model with variables of ethnicity, gender, disability, 
age, high school origin, high school education, high school GPA, years English, and grade in last 
English class can significantly predict English placement (Nagelkerke R 2 = 0.232). Further logistic 
models tested interactions between variables used to predict English placement. For ethnicity, there 
were significant interactions with high school GPA and grade in last English class (Nagelkerke R 2 = 
0.390). For gender, there were significant interactions with high school origin, high school degree, 
high school GPA, and grade in last English class (Nagelkerke R 2 = 0.197). For learning disability 
status, there were significant interactions with high school origin, high school GPA, and grade in last 
English class (Nagelkerke R 2 = 0.193). For age, there were significant interactions with high school 
origin, high school degree, high school GPA, and grade in last English class (Nagelkerke R 2 = 
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0.173). These relations between educational background and demographics suggest that the 
disproportionate impact observed in placements may be due to factors present prior to a student’s 
appearance at Cabrillo. 

Further research to identify the cause of these impacts and strategies to reduce these impacts will 
continue. In the meantime, Cabrillo College does offer a substantial set of matriculation services to 
support students such as our acclaimed Tutorials Center, Disabled Students Services, Fast Track to 
Work office for CalWORKs students, EOPS program for historically underrepresented students, 
and MESA center for economically disadvantaged and first generation college students. 



Table 5. Ethnicity of English Assessees from Fall 1994 to Fall 2000. 



Ethnicity 




Basic 

English 


English 100 


Transfer 

English 


Total 


African-American 


n 


80 


238 


50 


368 




% 


21.7 


64.7 


13.6 


100 


Asian 


n 


85 


302 


99 


486 




% 


17.5 


62.1 


20.4 


100 


Filipino 


n 


61 


157 


33 


251 




% 


24.3 


62.5 


13.1 


100 


Hispanic/Latino 


n 


1621 


2372 


340 


4333 




% 


37.4 


54.7 


7.8 


100 


Native American 


n 


39 


203 


61 


303 




% 


12.9 


67.0 


20.1 


100 


Other 


n 


126 


582 


221 


929 




% 


13.6 


62.6 


23.8 


100 


Pacific Islander 


n 


6 


55 


19 


80 




% 


7.5 


68.8 


23.8 


100 


White 


n 


919 


6695 


2785 


10399 




% 


8.8 


64.4 


26.8 


100 


Unknown 


n 


362 


666 


221 


1249 




% 


29.0 


53.3 


17.7 


100 


Total 


n 


3299 


11270 


3829 


18398 




% 


17.9 


61.3 


20.8 


100 



Chi-Square 2126.8 

df 16 

p < 0.0005 

Cramer’s V 0.240 
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Table 6. Gender of English Assessees from Fall 1994 to Fall 2000. 







Basic 

English 


English 100 


Transfer 

English 


Total 


Male 


n 


1752 


5172 


1307 


8231 




% 


21.3 


62.8 


15.9 


100 


Female 


n 


1321 


5723 


2404 


9448 




% 


14.0 


60.6 


25.4 


100 


Unknown 


n 


226 


375 


118 


719 




% 


31-4 


52.2 


16.4 


100 


Total 


n 


3299 


11270 


3829 


18398 




% 


17.9 


61.3 


20.8 


100 



Chi-Square 421.1 

df 4 

p < 0.0005 

Cramer's V 0.107 



Table 7. Learning Disability Status of English Assessees from Fall 1994 to Fall 2000. 







Basic 

English 


English 100 


Transfer 

English 


Total 


Learning 


n 


296 


510 


66 


872 


Disabled 

Not 


% 


33.9 


58.5 


7.6 


100 


Learning 


n 


2530 


9878 


3501 


15909 


Disabled 


% 


15.9 


62.1 


22.0 


100 


Unknown 


n 


473 


882 


262 


1617 




% 


29.3 


54.5 


16.2 


100 


Total 


n 


3299 


11270 


3829 


18398 




% 


17.9 


61.3 


20.8 


100 



Chi-Square 392.4 

df 4 

p < 0.0005 

Cramer's V 0.103 



Table 8. Age Descriptives of English Assessees. 





Median 


Mean 


sd 


N 


Basic English 


19 


22.0 a 


7.2 


2635 


English 100 


18 


21.7 a 


7.3 


9872 


Transfer English 


19 


23.3 b 


9.3 


3449 



Note: Means with different subscripts are significandy different from each other. 
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Table 9. ESL Placement by Financial Aid Application Submission as a Socioeconomic Indicator. 





Financial Aid 


Placement 


Did not apply 


Applied 


Basic English 


45% 


55% 


English 100 


56% 


44% 


Transfer English 


60% 


40% 


Chi-square 


195.0 




df 


3 




P 


< 0.0005 





Bias 

15 individuals were asked to review the writing prompts and state if they felt any of them were 
biased and if so to explain why. Demographics of respondents generally represented a variety of 
groups found at CabriJlo College (Table 10). Two-thirds of respondents did not indicate any bias 
concerns with the prompts. None of the prompts were said to be biased in and of themselves but 
there were concerns about possible biases of the part of the reader. For example, the 4 th prompt 
asked about 'Values, expectations, or beliefs”, which some respondents felt could result in the essay 
readers judging the student on their values. Fortunately, Cabrillo readers are trained to evaluate on 
the use of language rather than the content of the essay. Further, at least two different readers score 
the essays, which helps to minimize the bias of a single judge. Finally, assessees are allowed to 
choose between two different prompts so that fear of being negatively judged on one prompt can be 
avoided by choosing another prompt. While these safeguards address the concerns expressed, I 
recommend that future prompts created by the English department avoid asking about beliefs or 
values. 




9 



7 



Table 10. Demographics of Bias Survey Respondents 



Age 


Min 


20 


Max 


50 


Mean 


30.5 


sd 


8.7 


Gender 


Frequency 


Female 


ii 


Male 


4 


Ethnicity 


Frequency 


African-American 


i 


Asian 


2 


La tino / Hi sp anic 


6 


White 


6 


Disability 


Frequency 


Not Disabled 


ii 


Disabled 


4 


Sexual Orientation 


Frequency 


Bisexual 


2 


Heterosexual 


11 


Homosexual 


1 


Unknown 


1 


Religion 


Frequency 


Buddhist 


2 


Christian 


5 


Pagan/Wiccan 


1 


None 


4 


Other 


2 


Unknown 


1 


Language 


Frequency 


English 


8 


Russian 


1 


Serbian 


1 


Spanish 


4 


Vietnamese 


1 



Conclusion 

The English Holistic Assessment generally appears to meet or exceed minimum standards for 
use especially if problematic prompts are replaced. In addition, Cabrillo should continue to research 
possible causes of differential placement and continue student support services for historically 
underrepresented or disadvantaged students. 
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